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ABSTRACT 

This paper describes the results of a qualitative analysis of video captured during a dual indoor/outdoor Augmented 
Reality experience. Augmented Reality is the layering of virtual information on top of the physical world. This Augmented 
Reality experience asked students to interact with the San Diego Museum of Art and the Botanical Gardens in San Diego's 
Balboa Park. Pairs of students were videotaped as they progressed through the experience and the results of video 
analysis indicated that student engagement and collaboration was relatively high even though there were several 
technological difficulties - particularly present during the outdoor portion of the experience. 
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INTRODUCTION 

This paper is intended to describe videotape analysis of an 
Augmented Realily (AR) curriculum to determine the level 
of student engagement engendered by the experience. 
Augmented realily is defined as "games played in the real 
world with the support of digital devices (PDAs, cellphones) 
that create a fictional layer on top of the real world context" 
(Squire 8c Jan, 2007, p. 6). It has been identified in both the 
2010 and 2011 Horizon's ReportsWork at Harvard, MIT and 
the University of Wisconsin has shown promise in exploring 
the academic impacts of AR, and have also started to 
explore the issues of engagement and student affect 
associated with AR curricula (see, for example, 
Klopfer,Yoon,8c Perry, 2005; Squire 8c Jan, 2007; Klopfer 8c 
Squire, 2008; Dunleavy, Mitchell, 8c Dede, 2009; O'Shea, 
Mitchell, Johnston, 8c Dede, 2009; and O'Shea, Kaur, 
Amaechi, 8c Dede, under review). These initial efforts, 
however, depended upon instruments that were either 
leveraged from other fields or were developed internally to 
judge the engagement levels of students in the 
augmented realily gamespace. 

The AR curricula designed at Harvard, MIT and the University 
of Wisconsin were placed in outdoor settings, and used 
GPS-enabled handheld computers. In these instances, 
students were presented with a map of their physical 


surroundings on the handheld device and as they moved 
around the physical space an avatar would move around 
the map. Their tasks revolved around finding and interacting 
with virtual characters and objects that were found on the 
map. Conversely, there have been indoor efforts at AR 
experiences that used some other means of triggering these 
events, such as RFID tags (see, for example, work by 
Papadimitriou, Komis, Tselios, 8c Avouris, 2006). 

This paper will describe an AR curriculum that incorporates 
both indoor and outdoor elements. This educational 
experience takes place through the School in the Park (SITP), 
which "is an innovative program that shifts the location of 
'school' from a traditional classroom setting in an inner-city 
school, to the resources and educational opportunities 
available at museums in Balboa Park. A relevant 
instructional environment is created as hands-on learning 
challenges students to become active participants in their 
own education" (School in the Park, 2009). Elementary 
students are brought from their home schools for eight 
weeks a semester to participate in educational 
experiences in the museums located in Balboa Park. The AR 
curriculum described in this paper took place in the San 
Diego Museum of Art (SDMA) and the Botanical Gardens. 

The Crane Augmented Reality Curriculum 

This AR experience was called The Crane, and was based 
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on a Chinese Folktale called The Lord of the Cranes (Chen, 
2000). In this folktale, a Chinese mystic decided to leave his 
mountaintop retreat and visit a village in the valley below. 
As part of his journey, he dressed as a street-beggar to learn 
how people would treat him. There was only one person 
who showed compassion to the beggar/mystic, an 
innkeeper who provided him with food and lodging. In 
response to the innkeeper's kindness, the mystic paints a 
magical mural of cranes on the wall of his restaurant. 
Whenever there was happiness in the restaurant (for 
example, clapping or laughter) the cranes in the mural 
would dance for the customers. This magical mural 
became very famous and drew large crowds to the 
restaurant, thus making the innkeeper very wealthy and 
famous. The mystic asked only that the innkeeper continue 
to show the same kindness and dignity to other people in 
return for his mural. 

The indoor aspect of the AR experience asked students to 
interact with the museum's artifacts in order to prove 
themselves worthy of getting to the restaurant and seeing 
the dancing cranes. In order to do this, students selected 
one of four roles (a Traveler, a Potter, a Weaver, or an 
Emperor). Based on their role, they interacted with the 
artifacts differently. These interactions were triggered 
through QR codes (Figure 1) that were placed near the 
artifacts involved with the experience. Students would 
trigger the QR code with a QR reader on their cell phone 
(which ran the Google Android OS and were provided to 
the students for this experience), and based upon which 
role they had selected, they would be provided with a role- 
specific task to accomplish. For example, each student 
would be shown four images of items within the museum 
exhibit and be told to trigger the QR code next to the item 
that was must appropriate for their role (e.g. the Emperor 
role had to trigger a pair of decorative fingernail protectors 



Figure 1. QR Code that Triggers the Beginning 
of the Indoor AR Portion 


while the traveler had to trigger a water flask). If they were 
unsuccessful, they would be asked to select again while 
successful completion of these tasks provided each 
student with clues to their next task. At the end, after 
successfully navigating through all of the tasks, they were 
granted access to a virtual restaurant where they could see 
the cranes dancing in the mural. 


The outdoor aspect of the AR experience took place a 
couple days later and asked students to interact with the 
physical environment of the Botanical Garden, which is 
located right next door to the Museum of Art. In this case, 
the students would again select their role, although they did 
not have to take on the same role that they had for the 
indoor portion, and their task was to help plan a Chinese 
New Year celebration to take place at the innkeeper's 
restaurant (for example, the Weaver was asked to explore 
the surroundings to find inspiration for a gown that the 
Emperor had asked them to create for the ceremony). 
Students did this by interacting with virtual objects and 
characters through the Layar Augmented Reality app on 
their cell phone (Figure 2). The location and approximate 
distance to individual interactions was displayed on the 
device, and when used as a "viewfinder" the students 
would be able to see where they needed to go next as a 
virtual icon on the screen. Successful completion of each 



Figure 2. Screen Shot of Layar Augmented 
Reality App at Botanical Gardens 
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individual task provided the students with clues to their next 
task, and at the end of the experience the students would 
see a video of a Chinese New Year celebration. 

Methodology 

In order to evaluate the engagement levels for students, 
evaluators shadowed several student pairings as they 
participated in the ARE (both indoor and outdoor) and 
videotaped these interactions. Researchers followed 
specific student parings in an effort to witness engagement 
throughout the entire Crane ARE. Additionally researchers 
randomly videotaped student pairings throughout the ARE 
experience (both indoor and outdoor). 

A basic content analysis was then conducted on the entire 
video dataset (6,953 seconds). The videotape was 
analyzed using an a priori qualitative coding scheme that 
was originally developed through the Handheld 
Augmented Reality Project specifically for Augmented 
Reality implementations. This original coding scheme was 
adapted to align with this particular ARE project. Code 
adaptations were done through an iterative process 
including the modification of coding categories and 
coding descriptions. These modifications were done to 
provide a basic validation of the coding scheme or to show 
that more than one coder could use the coding scheme 
for measurement and get similar results. 

Intercoder reliability 

The goal of any content analysis is to record relatively 
objective (or at least intersubjective) characteristics of 
messages (Neuendorf, 2002). Given this goal it was essential 
to validate the coding scheme, establishing clarify 
surrounding its definitions, so that multiple coders identified 
the same messages within the videos being analyzed. As was 
stated above, the coding scheme was modified through an 
iterative process. Intercoder reliability was a driving force 
behind the code modification. Using the adapted coding 
scheme (see appendix A) acceptable intercoder reliability 
measures (Kappa values above .80) were achieved for all 
variables (codes) within our coding scheme. The coding 
scheme and the corresponding intercoder reliability 
measures have been included in Table 1. The full description 
of each of these codes, along with indicating actions can be 
found in Appendix A. 


Code (variable) 

Intercoder reliability (Kappa) 

Technology issues (engagement) 

.82 

Technology (engagement) 

.85 

Curriculum issues (engagement) 

.86 

Collaboration (general) 

.92 

Non-collaborative (engagement) 

.80 

Disengagement 

.93 

Instructor Interactions 

.95 


Table 1. A priori coding scheme with corresponding 
intercoder reliability (Kappa) values 


Findings 

Coding was done using NVivo, which allowed for the 
exploration of the durations involved with each code. The 
following section will present data associated with the ARE 
implementation as a whole, and then data associated 
only with the outdoor and indoor portions of the ARE. As 
student pairings could, potentially, fall into more than one 
code at a time (for example, through performing actions 
that indicate both collaboration and engagement at the 
same time) or into no category at given times, the data will 
not add up to 100 percent of the available time. 

ARE (all videos) 

Overall we analyzed 6,514 minutes (one hour, 38 minutes, 
34 seconds) of student activity in the Crane ARE. As a 
percentage of total video duration analyzed 71% of the 
video was shot indoors and 29% during the outdoor activity. 
This disparity, between indoor and outdoor video capture, 
was unintentional and is addressed in the analysis below. 

Table 2 shows that the curriculum issues (collaboration) 
code was used most often (36 percent of the total coding 
references) and encapsulated the most time (41 percent 
of the overall video duration). This indicates that the 



Table 2. Coding references, number of sources, and duration 
for the overall ARE (both indoor and outdoor) 


32 


i-monoger’s Journal on School Educational Technology, Vol. 7 • No. 1 • June - August 2011 









RESEARCH PAPERS 


students spent a considerable amount of time engaged 
collaboratively in solving curriculum issues will playing the 
Crane ARE. Furthermore, students were engaged with the 
technology, 23% of the overall codes were associated with 
technology (collaboration) indicating that the students 
were engaged and worked together with the technology. 
Although the technology seemed to be central in the 
experience (ranked second and referenced 95 times) the 
time students spent specifically engaging with the 
technology ranked forth (7.34 minutes), indicating that the 
students spent more time interacting with the curriculum 
(curriculum issues), instructors (instructor interactions), and 
engaged with the Crane ARE in non-collaborative ways 
(non-collaborative), such as watching videos. 

Alternatively, relatively low percentages of the time 
indicated disengagement (1.61 minutes) and issues with 
the technology (1.11 minutes). It's important to point out 
that this particular ARE was designed to minimize 
competition, so it is not surprising that there would be very 
little action indicating this was taking place. 

Indoor and outdoor ARE 

As is described above the indoor ARE was a somewhat 
different experience than the outdoor ARE. Therefore, the 
authors disaggregate the videos in this section and 
analyzed them separately in an attempt to understand the 
differences. There was considerably more video shot 
during the indoor ARE (76 minutes) than the outdoor ARE (32 
minutes). This discrepancy occurred not out of design but 
from basic logistics of the researchers' availability and time 
on site to capture video. In order to make comparisons 
between the indoor and outdoor data two tables were 
generated. Table 3 compares raw coding references and 
Table 4 uses a weighted average based on the overall 
duration of video included for each separate ARE 
experience (indoor 76 minutes versus outdoor 32 minutes). 

Table 1 indicates that students spent a considerable 
amount of time collaborating around curriculum issues in 
the indoor (106 references) versus the outdoor ARE (47 
references). However, in the weighted comparison (Table 
4) the coded references, normalized on duration, seem to 
be closer to equal. Continuing with these comparisons 
technology engagement was apparently greater during 


the indoor ARE experience, this difference is apparent even 
in the weighted scores (Table 4). This indicates that the 
students spent more time collaborating with the 
technology indoors than out. 

Although the raw coded references indicate that instructor 
interactions were similar (Table 3), the weighted scores 
(Table 4) indicate that instructors provided more assistance 
and direction overall during the outdoor ARE. Furthermore, 
students' actions were coded as non-collaborative, 
walking with a purpose and watching videos, more during 
the indoor ARE experience. Technology issues, although 
relatively low, were coded as more frequent during the 
indoor ARE. Finally, both general collaboration and 
disengagement were overall relatively low and equal 
between the indoor and outdoor experience. 

Analysis 

Reflecting on the weighted scores, observed (coded) 
instructor interactions were proportionally higher, and 
technology collaboration was proportionally lower during 


120 



Table 3. Coding references outdoors 
versus the indoors ARE experience 



Curriculum issues Technology Instructor Interactions Non-Collaborative Disengagement General (collaboration) Technology Issues 
(collaboration) (collaboration) (collaboration) (collaboration) 

■ Outdoor 1.516129032 0.516129032 1.225806452 0.032258065 0.258064516 0.258064516 0.032258065 

■ Indoor 1.394736842 1.039473684 0.421052632 0.605263158 0.197368421 0.197368421 0.223684211 

Table 4. Weighted average by duration of video included 
outdoor versus the indoor ARE experience 
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the outdoor ARE. Observations shows that students 
confronted significantly more technology troubles when 
participating in the outdoor ARE and these troubles impact 
the types of instructor interactions that students had with 
teachers. While most of these interactions during the indoor 
ARE take the form of large group direction-giving, the 
interactions with teachers outdoors tended to be one-to- 
one and focused on technological issues. Furthermore, as 
is indicated, observed and coded technology issues were 
higher during the indoor ARE indicating that students also 
encountered technical issues indoors but were able to 
solve these issues alone without instructor interaction (see 
definition of technical issues for clarification). 

Interesting, coded observations for non-collaborative 
activity was proportionally higher in the indoor ARE 
indicating that the experience required more individual 
actions, and disengagement was slightly higher during the 
outdoor ARE but less so than the researchers anticipated. 
Despite the technical issues encountered during the 
outdoor ARE on average the students spent equal time 
engaged with the curriculum with the outdoor ARE being 
slightly higher. This is encouraging, indicating that even 
though technical issues may be encountered students 
persisted and continued on with discussions and readings 
related to the academic content of the ARE. 

There are several important findings from this analysis. First, 
the technology troubles that students confronted when 
participating in the outdoor ARE impacted the types of 
instructor interactions that students had with teachers. 
While most of the interactions during the indoor ARE took the 
form of large group direction-giving, the interactions with 
teachers outdoors tended to be one-to-one and focused 
on technological issues. It was observed that students 
became much more frustrated with the technology during 
the outdoor ARE, gave up because of the difficulties, and 
resorted to asking a teacher for help. This resulted in an 
increase in the frequency of instructor interaction coding. 
As well, and related, students interacting together to solve 
technology issues (coded as technology issues) or using the 
technology together went down considerably during the 
outdoor ARE. The researchers believe that this may be due 
to the fact that as frustrations with the technology increased 


students seemed to take less time troubleshooting the 
problems together and went directly to the teacher for 
assistance. Despite the technical difficulties encountered 
in the outdoor ARE, the weighted averages for 
collaboration around the curriculum [curriculum issues 
coding) were almost equal. This indicates that although 
students encountered technical issue they found help, 
persisted and engaged in the unique learning process. In a 
separate study, significant gains were established in a pre¬ 
test-post-test analysis indicating that students improved 
their overall comprehension and that their performance on 
the test measure improved, an indication of learning 
(O'Shea 8c Folkestad, 2010). 

Stated simply, there were technological concerns with the 
outdoor ARE that were not necessarily seen in the indoor 
ARE. A series of informal "wrap-up" sessions with teachers 
and students indicated that these issues were well 
understood by the instructional staff, and mid-course 
corrections were taken to deal with these technical 
difficulties. Primarily these technical difficulties had to do 
with the layout of the Botanical Garden's structure and the 
scale at which the ARE was played. The Botanical Gardens 
are situated in a rectangular wooden lattice building, with 
entrances at the mid-point of the longest side. This meant 
that students had to enter the building in the middle and 
then progress to either end. Ideally, the triggering 
mechanisms in the Layar software would have activated 
the virtual elements of the ARE whenever a student entered 
a circular area around a GPS coordinate, however, the 
building's layout meant that the students would have had 
to walk through these triggering locations out of order. 
Figure 3 shows the dimensions and layout of the Botanical 
Gardens (white rectangle), and the arrangements of the 



Figure 3. Triggering Map for ARE Events in the Botanical Gardens 
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triggering events (yellow circles). As can be seen, students 
would have to walkthrough the area for event number 4 to 
get to any of the earlier events. Since the Layar technology 
did not allow for these events to trigger each other in 
sequence (e.g. have event 1 provide access to event 2, 
etc), it was necessary to have all events available from the 
start of the ARE. Even if event 1 were located at the 
entrance to the building, the size of the circular areas 
around the GPS coordinates would have meant that 
students would have had to walk through events out of 
order. The linear triggering of the events would only have 
been possible if the entrances had been situated at either 
end of the building. 

Secondly, the GPS accuracy on the devices was 
problematic based on the scale at which the ARE was 
played. The fact that the ARE was played within the 
Botanical Gardens constrained the event to an area 
approximately 300 by 200 feet large. At this scale, the GPS 
was not accurate enough to consistently provide reliable 
locations for individual triggering locations. After all, with a 
consistent GPS error of approximately 30 feet, the devices 
themselves could think they were actually outside the 
building, even though they were inside. This meant that 
events could be triggered outside of the Botanical Gardens 
rather than inside the structure. Playing the ARE with the 
same events but using a playing area double or triple the 
size would have minimized the impact of this GPS error. 

Having explained these difficulties, it is important to 
remember that the level of engagement in the outdoor 
ARE was still very high. Although lower than that seen in 
indoor video, the analysis still showed very low levels of 
disengagement during both indoor and outdoor 
experiences. 

Conclusion 

In conclusion, the functionality of the ARE is highly 
dependent upon how the technology and curriculum 
interact. The video of students participating in the 
experiences indicated that the functionality of the indoor 
ARE worked more smoothly than the outdoor, but also 
showed that students were engaged in both settings and 
with the content. Also, the video indicated that students 
worked collaboratively, although efforts could be made to 


facilitate and make communication between partners 
more robust. 

Aside from some pressing technological problems, which 
should mitigate as the technology becomes more robust 
and through curricular tweaking, the largest take away from 
this evaluation was the level of engagement; the 
researchers found that these kinds of experiences can be 
highly engaging for students and that commitment can 
provide benefits to the learner. Of course, the primary 
objective is to have the technology be engaging, not 
creating a significant hurdle that the students and teachers 
have to overcome, while having the students engaged in 
learning about their assigned topic. The results from this 
study indicate that even though technical difficulties were 
encountered, engagement with the curriculum was still 
very high. 

In addition, the refined coding scheme itself and the 
established intercoder reliabilities are important outcomes 
of this study. In post study debriefings, the researchers 
acknowledge that these codes provide a significant 
foundation for future work and analysis. However, the codes 
may need to be modified and adapted based on context 
of the ARE under evaluation. 

Looking ahead, the researchers believe that it is important 
to think strategically about the use of AREs. It is unclear how 
much of the obvious engagement is based on the inherent 
benefits of augmented reality and how much is based on 
the novelty of using GPS enabled handheld devices. This is 
a intriguing area for future longitudinal study. Having said 
that, however, the fact that the students are engaged with 
the content and that the entire SDMA/SITP curriculum is 
helping students to learn contentfO'Shea 8c Folkestad, 
2010) might indicate that an activity that engages students 
based on the novelty effect isn't necessarily a bad thing. 

Appendix A- Engagement coding scheme 
Technology issues (collaboration] 

Indicate actions between students in an effort to overcome 
technical difficulties or to use the technology. Actions fitting 
this code include things such as: 

• Two students talking about program not working, 
loading (code the duration of the discussion) 
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• Students asking other students for help (code the 
duration of the interaction) 

• Students asking group members where things were on 
the handheld / game (code the duration of the 
interaction) 

Technology (collaboration) 

These are actions between students that indicated 
collaborative use of the technology 

• Two students touching the handheld device at the 
same time (code the duration of the combined touch 
only) 

• Two students passing the handheld device between 
one another (code 10 seconds only) 

• One student touching or steadying the hand of the 
student holding the handheld device (cod the 
duration of the student-to-student contact) 

Curriculum Issues (collaboration) 

This would indicate actions between students to share or 
solve curriculum issues. Actions fitting this code include 
things such as: 

• One student reading to the other student (code the 
duration of the reading only) 

• One student point at an object within the curriculum - 
not pointing direction to move or go (code the 
duration of the pointing) 

• One student asking a question about the curriculum 
(code the duration of the question and answer session 

• Discussing what to do related to the curriculum (code 
the duration of the conversation) 

• One student asking the other a question about what to 
write (code the duration of the question and answer 
session) 

General (collaboration) 

This would indicate actions between students to 
collaborate on other issues besides technology and 
curriculum. Actions fitting this code include things such as: 

• One student pointing the direction to the next location 
(code the duration of the point only) 

• One student pulling the other student by the shirt 


sleeve, or jacket to provide direction (code the 
duration of the pulling action only) 

• A student taking leadership and directing other 
students (code the duration only) 

Non-coUaborative 

This would indicate non collaborative actions 
demonstrating engagement with the content. Actions 
fitting this code include things such as: 

• Walking with a purpose (code 10 seconds only) 

• Watching the handheld device at the same time, 
(code the duration of the watching) 

Disengagement 

This would indicate actions by students demonstrating 
disinterest with the content or technology. Actions fitting this 
code would include: 

• Student walking away from student with devise (code 
the duration of separation only) 

• Student not listening to their partner or becoming 
distracted (code the duration of the separation only) 

• Student attempting to distract partner (code the 
duration of action only) 

• Students complaining about the activity (code the 
duration of the compliant only) 

• Students making comments such as: "I'm so bored." 
(code the duration of the comments only) 

Instructor Interaction (collaboration) 

This would be present when the teacher or other 
instructional staff interacting with the students for 
educational or technical purposes. Actions fitting this code 
would include: 

• Teacher giving instructions (code the duration) 

• Teacher asks how group is doing (code the duration of 
the conversation) 

• Students 'giving up 1 on the technology and asking an 
instructor for help (code the duration of the time until 
they get instructor assistance). 
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